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ABSTRACT: This paper presents a conceptual approach and a Web-based service that aim at 
supporting self-regulated learning in virtual environments. The conceptual approach consists of 
four components: 1) a self-regulated learning model for supporting a learner-centred learning 
process, 2) a psychological model for facilitating competence-based personalization and 
knowledge assessment, 3) an open learner model approach for visual interaction and feedback, 
and 4) a learning analytics approach for capturing relevant learner information required by the 
other components. The Web-based service provides a technical implementation of the 
conceptual approach, as well as a linkage to existing virtual environments used for learning 
purposes. The approach and service have been evaluated in user studies in university courses on 
computer science to demonstrate the usefulness of the overall approach and to get an 
understanding of some limitations. 

KEYWORDS: Self-regulated learning, competence-based knowledge space theory, learning 
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1 INTRODUCTION 

This paper presents an approach and a service that support learners to learn in a self-regulated way. It 
has been shown in many studies that self-regulated learning (SRL) has many positive effects on the 
learning process, such as better learning in terms of being able to monitor, evaluate, and plan the 
learning process effectively, having better time-and-effort management, and demonstrating higher 
motivation for learning (e.g., Pintrich, 2000; Pintrich & De Groot, 1990; Zimmerman, 2008). Hence, 
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learners able to learn in a self-regulated way can achieve better results. However, there are also studies 
suggesting that many learners have problems with this way of learning. Taking over control of one's own 
learning process and applying metacognitive strategies (i.e., monitoring, evaluating, and planning the 
learning) require specific metacognitive skills that not all students have (Mikroyannidis et al., 2013). 
Therefore, learners often need guidance on different levels for learning in a self-regulated manner (Law 
et al., 2012). 

Though many technology-enhanced learning (TEL) solutions have evolved over the last decades, very 
few of them support SRL. Learning Management Systems (LMS) often used in educational settings, such 
as Moodle or Sakai, have become very popular (Paulsen, 2003). They focus primarily on distributing 
learning content, organizing the learning processes, and serving as an interface between learner and 
teacher. However, courses designed in LMS usually do not give learners much freedom, but rather 
provide a predefined learning trajectory. In contrast, Personal Learning Environments (PLE) (Henri, 
Charlier, & Limpens, 2008) strive for a more natural, learner-centric approach characterized by the 
freedom that individual learners have to select and control the services and tools they use. While this 
approach allows better opportunities for self-regulated learners, it still lacks guidance and help for 
learners with poor SRL skills. 

Another category of TEL solutions include adaptive systems and intelligent tutoring systems that aim to 
tailor their content and behaviour to the needs and preferences of learners (Brusilovsky, Kobsa, & Nejdl, 
2007). These systems are supposed to make learning more efficient by guiding the learner through the 
learning process. However, a self-regulated learner should not give control to a system, but take over 
the control on his own. The most promising approach towards SRL support is the concept of Open 
Learner Models (OLM). While adaptive systems make use of learner information (e.g., knowledge state) 
by using a learner model to adapt a course or system behaviour, the OLM approach pursues the idea of 
displaying learner information by making the model open to the learner and letting him choose the next 
steps. The main aim of the OLM approach is to support the learner's reflection process by providing 
formative feedback on the learning process, which has a positive effect on the learning outcome (Bull & 
Kay, 2010). 

Having the OLM concept in mind, the approach and service presented in this paper aim at supporting 
SRL on a broader scale. In addition to reflection support, there is also support for planning, goal setting, 
self-monitoring, and self-evaluation. This is achieved by defining a self-regulated learning process that 
includes the aforementioned metacognitive activities. Individual tools with interactive visual interfaces 
for the diverse metacognitive support strategies provide technical support. On a conceptual level, this is 
achieved by combining the self-regulated learning approach with Competence-based Knowledge Space 
Theory (Albert & Lukas, 1999) and resulting personalization strategies. The technical infrastructure in 
the background manages learner models and domain models representing the subject domain. It also 
contains a learning analytics component for analyzing the learner's behaviour and providing visual 
information and recommendations. 
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This paper is structured as follows: the next section gives an overview of the relevant state of the art 
and theoretical background used for the conceptual approach, followed by research questions derived 
from a literature review. We then explain a new conceptual approach for SRL support, describe the 
technical design of the models and developed service, and explain the learner-centred approach with a 
use case example. An evaluation of this approach and service conducted in a university course on 
computer science is reported in the section following. Results, limitations, and opportunities are 
discussed in next. Finally, concluding remarks and future work is described in final section. 

2 THEORETICAL BACKGROUND AND RELATED WORK 

2.1 Self-regulated Learning 

From a psychological and pedagogical point of view, self-regulated learning is a complex field of 
research that combines motivational as well as cognitive and personality theories. Components of SRL 
are cognition, metacognition, motivation, affects, and volition (Kitsantas, 2002). According to 
Zimmerman (2002), students can be described as self-regulated to the degree that they are 
metacognitively, motivationally, and behaviourally active participants in their own learning process. To 
define students' learning as self-regulated, they have to use specific strategies for attaining their goals 
and their learning behaviour has to be based on self-efficacy perceptions. In self-regulated learning, 
learners are active and able to control, monitor, and regulate their cognition, motivational state, 
behaviour, and context. Furthermore, learners set goals and try to achieve them through progress 
monitoring. These self-regulatory activities are mediators between personal characteristics, contextual 
features, and actual performance in the learning process. In a meta-analysis conducted by Flattie (2009), 
it turned out that self-regulated learning is one of the most effective methods to reach learning goals. 

Zimmerman (2002) has developed a cyclic SRL model consisting of three phases: the forethought phase 
(i.e., goal setting or planning), the performance phase (i.e., self-observation processes), and the self¬ 
reflection phase (i.e., self-reflection processes). According to this model, learning performance and 
behaviour consist of both cognitive and metacognitive activities. Cognitive activities are related to 
dealing with subject domains; for example, acquiring domain knowledge through reading. 
Metacognitive activities are related to thinking about and regulating the cognitive activities; for 
example, making a plan about domain knowledge acquisition. 

Boekaerts (1999), who developed the three-layered SRL model, pursued a similar approach. This model 
deals with cognitive and metacognitive activities, as well as with goals and resources. The first layer 
deals with regulation of the self, which is related to the choice of goals and resources that learners 
make. The second layer focuses on the regulation of the learning process, which relates to the use of 
metacognitive skills to direct the learning process. The third layer describes the regulation of the 
processing modes, which describes the choice of cognitive strategies. 

According to Roberts & Erdos (1993), metacognition is a key concept in the study of cognition and it 
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plays an important role in the transfer of cognitive skills and in problem solving. Often the term 
metacognition is simply defined as "thinking about thinking" or "cognition of cognition" (Flavell, 1976). 
This means that metacognition can be understood as the competence of reflecting on a mental task 
critically and of organizing the relevant learning and thinking processes in an efficient and effective way. 
Treier (2004) describes metacognition by the sub-components of self-monitoring, self-observation, and 
self-regulation related to cognitive and information processing. The usage of metacognitive strategies is 
an essential component of self-regulated learning and is very important for flexibility and 
personalization (Efklides, 2009). 


A key aspect of SRL is the learner's use of different cognitive and metacognitive strategies with the aim 
of controlling and regulating their learning (Pintrich, 1999). These strategies relate to being effective in 
learning, being able to self-regulate and control cognition (learning about learning), and being effective 
in applying resource management strategies. Dabbagh & Kitsantas (2004) summarized six key processes 
essential for SRL: 

1. In the goal-setting process, the outcome of a learning process is defined and strategies are 
identified for how to reach these goals. Goal setting motivates the learner's choice of and attention 
to the relevant tasks and motivates the learner towards higher effort and higher persistence over 
the course of time (Zimmerman, 2002). Furthermore, goal setting influences learning through 
affective reactions; for example, higher self-satisfaction when goals are reached. 

2. Self-monitoring is defined as one's reflected attention to an aspect of behaviour that directs the 
learners' attention to the task and assists them in evaluating the outcomes of their efforts. Self¬ 
monitoring is important, because it helps learners attain their goals by adjusting their learning. 

3. Self-evaluation is the process by which the learner compares the learning outcome with his own 
goals. It fosters better skill acquisition, self-efficacy beliefs, intrinsic interest, and self-satisfaction 
about performance. 

4. Task strategies are defined as the processes of a learner who applies strategies that help reach his 
own goals. Studies indicate that students who applied strategies for learning had better 
performance than students who did not regularly apply them (Pintrich, 1990). 

5. Help-seeking is taking place if a learner identifies and calls upon outside resources, not only human, 
but also analogue and digital resources. 

6. Time management is the process by which learners manage their learning regarding time. Effective 
time budgeting highly correlates with academic achievement. 


Supporting SRL in the right way is a crucial factor. On the one hand, it means providing enough freedom 
for the learner, in order to stimulate motivation. Flowever, on the other hand, too much freedom may 
be overwhelming and appropriate guidance or even adaptation is usually needed to make the learning 
process effective and efficient. The concept of guidance and freedom is important, because it has been 
recognized that highly motivated learners attain better learning performance if they have more control 
over their learning and are more autonomous (Issing, 2002). On the other hand, some learners show 
difficulties in carrying out concrete metacognitive activities, such as planning, goal setting, monitoring, 
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evaluating, and as a result often perform less successfully than would be anticipated (Bannert, 2006). 
Such learners are in need of guidance. Furthermore, less motivated learners can also improve their 
performance if they receive more guidance. Keeping these reported findings in mind, individual support 
for learners should be tailored to suitable degrees of guidance and freedom. In this respect, the learner 
should be offered an optimal and balanced level of control and autonomy for his own learning process. 

Motivation is a highly relevant aspect for achieving good learning outcomes and for performing self- 
regulated learning activities. Winne & Hadwin (2008) showed the positive impact of motivation on 
student attention to the learning progress, on the progress itself, and on the experience of satisfaction 
and positive affect. For the use of self-regulated learning activities, a learner has to be motivated, as 
these activities require additional time and effort. Ryan & Deci (2000) describe intrinsic motivation as 
one of the most important aspects regarding learning, because it is the prototypical manifestation of the 
human tendency toward learning and creativity. Flowever, there is also a need for extrinsic motivation 
and especially a good balance between extrinsic and intrinsic motivation (Covington, 2000). 

2.2 Learning Analytics 

The Society for Learning Analytics Research (SoLAR) defines Learning Analytics (LA) 1 as "the 
measurement, collection, analysis and reporting of data about learners and their contexts, for purposes 
of understanding and optimizing learning and the environments in which it occurs." Siemens (2010) 
describes learning analytics as "the use of intelligent data, learner-produced data, and analysis models 
to discover information and social connections, and to predict and advise on learning." 

In the NCM Florizon Report, learning analytics is described as a rapidly developing trend in higher 
education, where learning is happening more and more within online and hybrid environments 
(Johnson, Adams Becker, Estrada, & Freeman, 2014). According to this report, learning analytics can 
potentially help transform education from a standard one-size-fits-all approach into responsive and 
flexible frameworks. Such frameworks are capable of adapting content and software behaviour to the 
needs of learners and provided tailored recommendations and visualizations. New kinds of visualizations 
and analytical reports are being developed to guide administrative bodies with empirical evidence 
because they help assess and improve the effectiveness of programs, schools, and entire school 
systems, which aids in the proper allocation of resources. 

Duval (2011) outlines several possibilities about how learning analytics concepts and methods can be 
put into practice: 1) goal-oriented visualizations to impact the further behaviour of the user; 2) technical 
infrastructure to model the captured data; 3) dashboard applications to give a visual overview of the 
collected data and often relate that data to other learners. The captured data can also be used for 
learning recommendations of resources, activities, and people. 


1 See http://solaresearch.org/ 
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In a learning analytics process model (Verbert, Duval, Klerkx, Govaerts, & Santos, 2013), four stages of 
personal information and their applications are distinguished: awareness, reflection, sensemaking, and 
impact. Awareness is concerned with data represented as activity streams, tabular representations, or 
other visualization types. The reflection stage focuses on the user who asks questions regarding the 
provided data in order to investigate their usefulness and relevance. The sensemaking stage is about 
answering the posed questions to obtain new insights. Finally, the impact stage relates to the induction 
of new meaning or behaviour change. While most learning analytics approaches target reflection 
support, focusing on learning analytics infrastructure helps learners develop learning dispositions and 
transferable competencies, such as critical curiosity or creativity (Buckingham Shum & Crick, 2012). 

2.3 Competence-based Knowledge Space Theory and Adaptation Approaches 

Individually adapting learning methods to suit the learner's characteristics has a big influence on 
performance (Issing, 2002). The importance of the adaptation to the learner's characteristics (also called 
personalization) has been shown in several studies. For example, adaptive subject material combined 
with adaptive styles of presentation supports students to improve their learning achievements and 
increases learning efficiency (Tseng, Chu, Flwang, & Tsai, 2008). Through a requirement analysis, it has 
been found that the learner's knowledge, goals and tasks, language, and interests are important factors 
of personalization approaches (Flover & Steiner, 2009). 


Knowledge Space Theory (KST) and its competence-based extensions (CbKST) are prominent examples 
of adaptation strategies grounded on theoretical framework (Flockemeyer, 2003). KST constitutes a 
psychological mathematical framework for both structuring knowledge domains and representing the 
knowledge of learners (Albert & Lukas, 1999). Due to dependencies between problems, prerequisite 
relations can be established. The knowledge state of a learner is identified with the subset of all 
problems this learner is capable of solving. By associating assessment problems with learning objects, a 
structure on learning objects can be established, which constitutes the basis for meaningful learning 
paths adapted to the learner's knowledge state. 

Competence-based Knowledge Space Theory (CbKST) incorporates psychological assumptions on 
underlying skills and competences required for solving specific problems (Korossy, 1997; Heller Steiner, 
Flockemeyer, & Albert, 2006). In this approach, competences are assigned to both learning objects 
(taught competences) and assessment items (tested competences). Similar to the knowledge state, a 
competence state can be defined consisting of a set of skills that the learner has available. Furthermore, 
there may also be relationships between competences modelled in a prerequisite relation structure. 
CbKST provides adaptive assessment algorithms for efficiently determining the learner's current 
knowledge and competence state, which builds the basis for personalization purposes. Based on this 
learner information, personalized learning paths can be created. Goal setting can be done by defining 
skills to be achieved (competence goal) or problems to be capable of solving. The competence gap to be 
closed during learning is represented by the skills that are part of the goal, but not part of the 
competence state of a learner. 
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From a broader perspective, adaptation approaches have a long tradition in educational settings. For 
example, Intelligent Tutoring Systems (ITS) were supposed to bring intelligence to computer-based 
instruction, especially in the knowledge of the subject domain, as well as the tutoring principles and 
methods of their application (Anderson, 1988). This led to the development of four basic components: 
the domain model, the student model, the tutoring model, and the user interface model. Adaptive 
hypermedia systems in educational contexts are based on these models and aim at personalizing the 
learning experience (Brusilovsky et al., 2007). Finally, Open Learner Model (OLM) approaches (Bull & 
Kay, 2010) go one step further and make use of these models for pedagogical reasons by visualizing and 
presenting the information to the learner (see also the introductory section of this paper). 

2.4 Summary and Resulting Research Questions 

This section has presented three prominent learning concepts successfully applied in technology- 
enhanced learning. Each of them covers a specific field in the learning domain. Self-regulated learning 
focuses on metacognition and on how learners take over responsibility of their own learning process. 
Competence-based Knowledge Space Theory focuses on how the learning process can be structured in 
terms of the subject domain. Learning Analytics has its strengths in observing the learner and creating 
benefit from these observations. Personalization strategies and Open Learner Models are employed by 
the latter two concepts. Flowever, each of them alone does not cover the full spectrum of the learning 
process. Learning is a complex process and ideally covers all these aspects; therefore, it makes sense to 
elaborate a solution that combines these learning concepts with their specific characteristics. An 
approach to integrating SRL and CbKST on a conceptual level was elaborated in previous work (Steiner, 
Nussbaumer, & Albert, 2009). 

The next section presents a solution that combines the ideas and methods of the presented learning 
concepts. This leads to a new solution approach that smoothly integrates the methods and 
characteristics of these learning concepts by focusing on the benefit for the learner. This endeavour can 
be expressed in three research questions: 

• Rl: Flow can learners be supported in following a self-regulated learning paradigm and how can we 
ensure that they achieve their learning goals on the domain level at the same time? 

• R2: Flow can Competence-based Knowledge Space Theory, Learning Analytics, personalization, and 
Open Learner Models be combined to design a holistic framework that supports self-regulated 
learning? 

• R3: Flow should a system be designed that implements this conceptual approach? 

3 CONCEPTUAL APPROACH AND TECHNOLOGY 
3.1 Psycho-Pedagogical Framework 

The central piece of the psycho-pedagogical framework is the domain model that formally covers the 
subject domain to be learned. Competence-based Knowledge Space Theory (CbKST) provides a 
methodology to structure a subject domain and also makes it usable for a technical application. The core 
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elements as used in this framework are learning objects, assessment items, and competences. 
Competences are structured through prerequisite relations, meaning that it can be assumed that, if 
competence A is a prerequisite of competence B, then a learner having competence B also has available 
competence A. Figure 1 depicts a prerequisite structure of five competences (A, B, C, D, E) in an acyclic 
directed graph. Competences below others are prerequisite for them, if connected through a path to 
them. This type of graph is also called Hasse diagram. 

In order to structure a subject domain, the first step is to define competences necessary to master it. 
These competences are then structured according to their prerequisite relations into a competence 
map. In addition to competences, learning objects are created that convey these competences and 
assessment items are defined that test these competences. Thus, relations between learning objects, 
assessment items, and competences are established. The set of all elements and their relations is called 
the domain model (see Figure 1). Such a domain model is usually created by a domain expert or teacher. 




Figure 1. The diagram depicts the domain model structure on the left site and an example of 
five competences with prerequisite relations between them on the right side. 


The user model is another core component of the framework. In general, the user model follows an 
overlay design (Brusilovsky et al., 2007) meaning that the user model elements are related to and 
defined by other models. In our case, the user model refers to the domain model and the tool 
interaction model (see below). The user model allows for defining information about the learner's 
knowledge, competences, goals, history, and used tools. It relates to both aspects of the domain model 
and aspects of the self-regulated learning, and thus connects them. Table 1 gives an overview of the 
information contained in the user model. 


Further important components of the psycho-pedagogical framework are the learning process and 
learning cycle. The learning process is a rather generic term that refers to all cognitive and 
metacognitive activities of the learner, as well as the interactions with the learning system. The learning 
process starts when the learner begins to learn and ends when the learner has finished learning 
(independent of the results). In order to operationalize the learning process, a self-regulated learning 
process model is defined that connects self-regulated learning concepts with elements of the domain 
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model. The learning cycle is related to the iterations within this process (see below). 


Table 1. Overview of the user model and the contained information. 


Category 

Information/Items 

Description 

Learning 

State 

Knowledge State 

The assessment items that a learner can solve 

Competence State 

The competences a learner has available; each time a learner solves (or 
fails to solve) an assessment item, the related competences are added to 
or removed from the competence state 

Competence Goal 

The competences that a learner wants to acquire 

Competence Gap 

The competences needed to achieve the competence goal 

Learning 

History/ 

Learning 

Behaviour 

Learning Objects 

The learning objects a learner has visited 

Assessment Items 

Answers to assessment items including the information if solved or failed 

Learning Tool 

The tools and their submenus that a learner has used 

Help 

The help information that a learner has requested (related to a tool) 


So far, the described concepts can be employed in different learning approaches. Traditional adaptive 
systems use domain and user models to create learning paths automatically through the learning 
objects and assessment items. Though such approaches are personalized, the learners just follow the 
paths without having alternative options. The other extreme would be an approach where all learning 
objects and assessment items are available and the learner is free to choose, but without any help, 
recommendations, or feedback from such a system. While the first type of approach might be efficient 
for weak learners by offering strict guidance, it does not provide the advantages of self-regulated 
learning. On the other hand, the second type of approach has its weaknesses, because it does not 
provide any support, which is also helpful for learners familiar with this type of learning. The goal of this 
framework is to establish an approach that offers freedom and support at the same time. 

In order to facilitate support and guidance in a technical environment that does not restrict the freedom 
of the learner, a self-regulated learning process model is defined. This model follows the ideas of the 
cyclic SRL model of Zimmerman (2002) and SRL activities defined by Dabbagh & Kitsantas (2004). It 
describes learning as a cyclic sequence of four main phases: 1) planning and goal setting, 2) using 
learning resources, 3) knowledge and competence assessment, and 4) reflecting on learning behaviour 
and progress. For each of these phases, a visual tool that supports the respective cognitive and 
metacognitive activities is provided. In the planning and goal-setting phase learners set their short-term 
goals, in order to plan what they want to learn next. This phase is mainly related to the metacognitive 
activity of planning. In the next phase, the learners make use of learning resources fitting to the selected 
goals, in order to attain related domain knowledge. Learning in this phase mainly happens on the 
cognitive level, but is also related to the metacognitive level of self-observation. Then the learners 
undergo a knowledge assessment regarding the recently used learning content and current learning 
goal. This phase is also on a cognitive level and related to the self-monitoring activity. Finally, learners 
should reflect on the activities and outcomes of the last phases. The current goal, the visited learning 
objects, and the assessment results are visually displayed. Reflecting on learning behaviour and progress 
targets the metacognitive activities of self-reflection and self-evaluation. A learning cycle is defined as 


ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 


109 













JOURNAL OF LEARNING ANALYTICS 


S8LAR 


(2015). A Competence-based Service for Supporting Self-Regulated Learning in Virtual Environments. Journal of Learning Analytics, 2(1), 101-133. 


iteration through these four phases. An overview of these phases and their relations to cognitive and 
metacognitive activities is shown in Table 2. 


Table 2. Overview on the cognitive and metacognitive learning activities in the SRL process. 


SRL phase and related tool 

Metacognitive activity 

Cognitive activity 

Planning and goal setting 

Planning 

Understanding the subject 
domain 

Using learning resources 

Self-monitoring 

Attaining domain knowledge 

Knowledge and competence assessment 

Self-monitoring 

Knowledge assessment 

Reflecting on learning behaviour and progress 

Self-reflection, self-evaluation 

Knowing the learning progress 


A system that features these phases by offering appropriate tools can be regarded as an environment 
that enables self-regulated learning. A major goal of this approach is to provide personalized scaffolds 
that assist learners in a self-regulated manner. These scaffolds are based on the domain model and the 
user model. Since the user model contains information on the individual learning history, the support 
can be adapted to the individual learner and thus be personalized. Personalized scaffolds are given to 
the learner in each SRL phase differently depending on the phase and related tool. For example, in the 
goal-setting phase the learner gets a visual representation of the competences, including visual clues of 
the current competence state, so that the goal-setting activity is guided by the prerequisite structure of 
the competences and the current competence state. More details on personal guidance, the user 
interface, and its tools are provided in the User Interface section below. 

Learning Analytics methods are employed by exploiting the user model data and presenting relevant 
information in a graphical way. This representation should stimulate the learner's reflection and 
motivation. Instead of operating with pure log data, high-level information (based on competences, 
learning objects, and assessment items) is presented to the learner. Since the user model holds the 
history data in time sequences, calculations must be performed to extract information as described 
above (reflection phase). An Open Learning Model approach is employed by displaying information in a 
graphical way that allows the learner to self-regulate his own path through the learning resources. 
Personalization is implemented by giving visual recommendations in terms of selecting next goals and 
choosing next learning objects and by offering assessment items depending on the previous learning 
behaviour. 

Summarizing the described approach, the framework consists of a domain model and a user model that 
cover the subject domain and provide the basis for supporting learning. The self-regulated learning 
process model allows the learner to navigate freely through the learning resources, but provides self- 
regulated learning scaffolding to help the learner follow the phases and related tools. 
Recommendations, visualizations, and monitoring mechanisms are used to provide personalized 
guidance. The overall approach is outlined in Figure 2. 
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Figure 2. Overview of the psycho-pedagogical framework. The outer circle shows the 
metacognitive activities and the arrows indicate that these are supported in different ways 

based on user and domain models. 

3.2 Technical Design and Compod Service 

A core part of the technical design is the Compod service created to support SRL based on CbKST and 
learning analytics concepts as outlined above. This Web-based service uses a REST interface for 
functionality. It provides access to the user and domain model, as well as to the recommendation and 
assessment features. Furthermore, the service includes monitoring and tracking functionality to capture 
learner interactions, analyzing and reasoning features to model the learner state and behaviour, and 
access to this information. In order to offer this functionality to the learner, a technical component with 
a user interface is connected to the service (see the User Interface section below). The overall technical 
design is outlined in Figure 3. 

The service holds and manages the domain models, which are represented in XML format and contain 
information regarding learning objects, assessment items, and competences, as well as the relations 
between these elements. Domain models can be added, removed, or edited. The REST interface exposes 
methods to retrieve whole domain models, but also offers reasoning on it. For example, learning objects 
related to a specific competence can be searched for and retrieved. 


The user model component storage holds the learner information captured by the tracking and 
monitoring component, which captures the interaction data from the user interface and stores it in the 
user model as an extended semantic triplet. While the semantic triplet has the form <subject, predicate. 
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object>, the extension adds contextual information about creation time and domain model. The subject 
is related to the learner who induced the information. The predicate describes the type of information; 
for example, learning object visited, assessment item solved, assessment item failed, current competence 
goal, or current competence state. The object contains the actual information according to the predicate 
type, which can be, for example, a learning object, an assessment item, or a competence. The domain 
model information is needed to specify the subject domain to which the triplet is related. For example, a 
student can learn two different subject domains at the same time, which has to be made 
distinguishable. This way of structuring user model information creates a flat structure that can be 
browsed and analyzed easily. 
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Figure 3. Overview of the technical architecture and main components. 


The assessment component manages the knowledge and competence assessment. It recommends 
assessment items based on the current competence goal and previous answered assessment items. 
Furthermore, it processes the answers to the assessment items and calculates the current competence 
state. The recommendation component selects learning objects fitting the current competence goal and 
competence state. It also takes into account previously visited learning objects. These components form 
the basis for the personalization approach and personalized guidance. 

The learning analytics component is responsible for analyzing the user model information and creating 
meaningful reports that help learners gain insight into their learning process and improve their learning. 
For example, it contains statistics on the visited learning objects and completed assessment items in 
relation to the competence goals. Furthermore, it includes information about how the competence state 
evolved over time throughout the learning process. This information is provided in XML format via the 
REST interface and is the basis for the visualizations in the user interface. 

3.3 User Interface 

This section describes a user interface designed to interact with the Compod service. This user interface 
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consists of four tools representing the four phases of the SRL process model. Each tool supports the 
cognitive and metacognitive activities of the respective phase. Switching between these tools can be 
done by clicking on the tool name on top of the user interface. The tool names are the catchwords Plan, 
Learn, Assess, and Reflect. Following the sequence of the phases is suggested but not mandatory. 

The first tool addresses the goal-setting activity (see Figure 4). This tool displays the competence map 
consisting of the competences of a domain model and the prerequisite relations between these 
competences. As overlay information, it also depicts the current competence state by drawing the 
contained competences as bigger green circles. Furthermore, it displays the current competence goal by 
drawing a red border around the circles. The user can add or remove competences by clicking on the 
respective ones. The prerequisite structure serves as guidance to navigate through the subject domain. 
The learner is free to choose any competences, but from a pedagogical point of view, it is meaningful to 
start with competences that have no other competence as prerequisite and then move up along the 
prerequisite relations. Thus, the prerequisite structure and the current competence state are scaffolds 
for the learner to choose goals and navigate through the subject domain. 

The second tool is used to browse through the learning objects (Figure 5). All learning objects are listed 
on the left side, while the recommended learning objects are painted with a red border. Visited learning 
objects are marked with a blue line on the right side. Learners are free to choose, but it is recommended 
by visual scaffolds to follow the learning objects associated with the competence goal chosen by the 
learner. 


(f' Compod 

Learning Service 

Information Search and Retreival - Unit 6 

test-an | Organise | Help 

Plan ]=> Leam|=> Assess => Reflect 


Views: Recommendations • History 



Figure 4. The goal-setting tool displays the competence map, the available competences (in 
green) and allows for selecting a competence goal (red border). 
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The assessment tool, the third tool, provides two options for assessment that can be freely chosen by 
the learner. The tool presents assessment items related either to the goal competences or to visited 
learning objects (Figure 5). Therefore, the learner can choose to use the goal-setting feature and do the 
assessment according to these goals or to omit the goal setting and do the assessment according to the 
visited learning objects. 
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Figure 5. The learning and assessment tools are displayed in the screenshots. 


The reflection tool presents the learning progress of the learner (Figure 6). It displays an overview of the 
learning progress over time, which is depicted as a green line representing the proportion of available 
competences after each learning cycle. The proportion is calculated as the number of available 
competences divided by the number of all competences multiplied by 100. Furthermore, it displays the 
proportion of correctly (green bars) and incorrectly (red bars) answered assessment questions. Absolute 
numbers are available to the learner through the tooltip. The tool can also display the number of visited 
learning objects in relation to the selected competence goal and assessment items, if the learner selects 
the "Learning Behaviour" view. 

3.4 Case Study 

From a self-regulated learning perspective, the proposed way of using this interface and thus the 
Compod system is to start with the goal-setting tool and then use the learning tool, the assessment tool, 
and the planning tool. This sequence should be reiterated until all competences have been acquired. For 
example, learner X adds three competences to her current learning goal. Then she navigates to the 
learning tool where she gets recommendations (highlighted with red borders) for learning objects that 
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teach these competences. The learner visits these learning objects and learns the content. After 
learning, she navigates to the assessment tool where she gets assessment items related to the three 
competences in her current goal. She answers these items and it turns out that one answer is wrong and 
the others are correct. This information is immediately shown by putting green borders on the questions 
with correct answers and a red border on the question with the wrong answer. The learner then 
navigates to the reflection tool where this first learning cycle (iteration) is shown. The average 
competence level (number of available competences divided by the number of all competences) is 
depicted as a progressing line. Further, the numbers of visited learning objects and results on the 
assessment items are shown for this first cycle. After completing this cycle, the learner navigates again 
to the planning phase. Now the competence map still shows the previous competence goal (highlighted 
with red circles), but also shows the available competences (green circles) because of mastering 
assessment items in the previous cycle. Therefore, the learner can modify her learning goal. For 
example, she will remove the available competences from the current goal, keeping the competence 
where she failed in the previous cycle and adding two new ones. The remainder of this cycle is 
processed as before. 


Compod Information Search and Retreival - Unit 6 test-an | Organise | Help 

Learning Service 


Plan] => Learn => Assess| =>|Reflect| Views: Learning Progress * Learning Behaviour 

Learning Progress 
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0 r- 
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O Reflect your learning process. In this view the score and results to questions are displayed. 

Figure 6. The reflection tool is displayed in the screenshot outlining the learning progress. 

Though the case described above follows the self-regulated learning process model, learners can still 
choose to follow different learning paths and navigation behaviour. For example, learner Y prefers not 
to navigate through this whole cycle all the time so she navigates to the learning tools and starts 
learning with some of the learning objects. After a while, she navigates to the assessment tool and gets 
questions related to the visited learning objects. She completes the assessment with mostly correct 
answers and one wrong item. She then navigates back to the learning tool where she can see which 
learning objects teach the missing competences (this is a special view in the learning tool). The learner 
iterates this combination of the learning and assessment phase several times. From time to time, she 
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also navigates to the goal-setting tool where she gets an overview of her current knowledge state 
(available competences). 


Both cases describe a self-regulated learning behaviour, though these two learners pursue different 
strategies. Both are in control over their learning process and freely choose their learning behaviour. 
While learner X explicitly sets current goals and accepts respective personalized scaffolding strategies, 
learner Y omits the explicit goal setting and choses individual learning objects directly and creates an 
individual learning path through the learning objects. Learner X explicitly uses the reflection tool for 
feedback, but learner Y accepts feedback from the goal-setting tool where available competences are 
highlighted. 

4 EVALUATION 

In order to demonstrate the usefulness and applicability of the presented approach and service, the 
Compod system with its user interface was applied in the context of a university course held at Graz 
University of Technology. The main purpose of this evaluation was to answer the following questions. 
While the research questions outlined in the Summary and Resulting Research Questions section led to 
the solution approach and related system implementation described above, these evaluation questions 
intend to test these solutions: 

• El: How useable and acceptable is the overall Compod system and its user interface? This question 
addresses two aspects: usability and user acceptance. Usability refers to the issue of whether the 
system allows the user to achieve a specific goal effectively, efficiently, and satisfactorily. 
Concerning user acceptance, the main question of interest is whether users find the system 
acceptable and intend to use it. Research has shown that although a system is technically sound, 
users often do not intend to use the system because they lack a positive attitude towards system's 
usage (e.g., Davis, 1986; Hirschheim, 2007). Thus, answering this question is an indicator of whether 
learners will adopt the self-regulated way of learning in future. 

• E2: Do learners accept and follow the learning approach provided by the Compod system? Do they 
adopt the self-regulated way of learning? This question refers to whether learners actually use the 
system and its different self-regulated learning functionalities. 

• E3: Does use of the Compod system and its underlying learning approach benefit the acquisition of 
domain knowledge? This evaluation question addresses the learning effectiveness that can be 
achieved through self-regulated learning with the Compod system. 

• E4: How do users feel about the visualizations (i.e., goal-setting tool and reflection tool) provided by 
the system? This question refers to users' perceived benefit of the visualizations provided by the 
system. This includes questions like whether the visualizations are understandable and suitable for 
their given tasks, or whether the visualization types help them to plan or reflect on their learning 
process in terms of learning effectiveness. 
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4.1 Method 

This section presents the method and results of the evaluation study, consisting of two evaluation 
rounds conducted in 2013 and 2014. We employed a descriptive evaluation study design with a rather 
formative character aimed at improving the software and the learning approach. In the following 
sections, the methodology of the evaluation study is first described, and then the analysis of the data 
collected is given. 

4.1.1 Setting 

The study was conducted in the context of a university course on "Information Search and Retrieval 
(ISR)" held at the Graz University of Technology, Austria, in November 2013 and November 2014. This 
university course is a typical mandatory computer science course in the Master's Program, which 
guarantees that enough students are available and motivated to participate in research studies. The 
whole lecture consists of different topics, such as Web Retrieval, Query Languages, or Information 
Retrieval Models. For one topic, namely "User Interface and Visualization" about multimedia 
information retrieval and user interfaces in information retrieval, the Compod system has been used. In 
order to use the Compod system and its functionalities adequately, the domain model was created by 
the authors. The domain model consisted of 26 competences structured through prerequisite relations 
(see Figure 4). Furthermore, 32 learning objects and 26 assessment items related to the competences 
were created. The learning objects included the content conveyed in the lecture on this unit in previous 
years. 

4.1.2 Participants 

The study was carried out with two groups of Master's students at two points in time. In the first group 
(winter 2013), 28 students (6 female, 22 male) took part and filled in the evaluation survey. The sample 
consists of students of Computer Science (20 students), Software Development and Business 
Management (5 students), and Telematics (3 students). Students indicated that they had a medium pre¬ 
knowledge of the subject taught. In the second group (winter 2014), 22 students (4 female, 18 male) 
participated in the study and completed the evaluation survey. Participants were students of Computer 
Science (18 students), Software Development and Business Management (2 students), and Telematics (2 
students). They indicated that they had an average knowledge in the field taught. 

4.1.3 Material 

In order to collect and analyze quantitative and qualitative data, a multi-method approach consisting of 
user model data sources and questionnaire data sources was applied in this study. 

User Model Data: For investigating how and in which way students use and apply the system and its 
underlying learning approach, data on the learning and navigational behaviour of students was recorded 
as useful descriptive information. Learning behaviour data contains information on the visited learning 
objects, the responded assessment items, selected goals, and the acquired competences. Navigational 
data contains log data recording user interaction with the Compod system, or more concretely, which 
tools were used in which sequence and with what frequency. 
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Questionnaire: An online survey was created consisting of five short questionnaires allowing for 
capturing quantitative as well as qualitative feedback from students. The survey was realized and 
administered using an online survey system. The following main aspects were addressed by the online 
survey: usability, user acceptance, learning approach and guidance, usability and benefit of the goal¬ 
setting tool, and usability and benefit of the reflection tool. 

For answering the first evaluation question referring to the general usability of the system (El), the 
System Usability Scale (SUS; Brooke, 1996) covering 10 items was used. With respect to user acceptance, 
which refers to the first evaluation question (El), a scale of three items covering the main aspects (i.e., 
perceived ease of use, perceived usefulness, and intention to use) according to the technology 
acceptance model (Davis, Bagozzi, & Warshaw 1989) was adapted. 


The aspects learning approach and guidance were captured by three items each. These newly created 
items ask for general level feedback on the usefulness and applicability of the learning approach used by 
the system. To collect qualitative feedback on these aspects, this section was completed with one open 
question where participants had the opportunity to give additional feedback and comments. Gathering 
data on this aspect allows for answering the second evaluation question (E2). 


In order to evaluate visualization types used for the planning (i.e., goal-setting tool) and reflection (i.e., 
reflection tool) phases and in order to answer the fourth evaluation question (E4), a questionnaire 
developed in order to evaluate visualizations in the context of digital libraries (Steiner et al., 2014) was 
adapted (see Table 3). Overall, the questionnaire consists of two scales: one (4 items) assessing usability, 
and the other (6 items) investigating the perceived benefit. Usability consists of the subscales suitability 
for the task and self-descriptiveness. The perceived benefit scale consists of the subscales 
metacognition, cognitive load, and learning effectiveness. At the end of the study, participants were 
asked to provide qualitative feedback on the strengths and weaknesses of the goal-setting tool and the 
reflection tool. 


Table 3. Subscales on usability (i.e., suitability for the task and self-descriptiveness) and 
perceived benefit (i.e., metacognition, cognitive load, and learning effectiveness) to evaluate 
the goal-setting tool as well as the reflection tool. 


Subscale 

Items 

Suitability for the Task 

1 find this visualization suitable for getting an overview of the current status in the learning 
process. 

1 think the visualization provides irrelevant information. 

Self-Descriptiveness 

It is easy to understand this visualization. 

1 find this visualization unnecessarily complex. 

Metacognition 

1 think this visualization can help learners reflect on their learning process. 

1 think this visualization supports learners in better planning their learning goals. 

Cognitive Load 

1 think interpreting this visualization would put additional cognitive effort on the learner. 

1 think this visualization is able to leverage the mental workload. 

Learning Effectiveness 

1 think this visualization can help learners in accomplishing their goals. 

1 think the use of this visualization will not make a difference for learning performance. 
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Each aspect (i.e., user acceptance, learning approach and guidance, goal-setting tool, and reflection tool) 
— except usability — covered by the survey was assessed with items or statements answered on a 
seven-point rating scale ranging from strongly disagree (1) to strongly agree (7). Negatively poled items 
have been recoded for further calculations. Then, for each aspect, a mean score averaging across the 
rating scale can be calculated with higher values indicating a better result. For assessing overall usability, 
we applied the SUS, including statements rated on a five-point scale ranging from strongly disagree (1) 
to strongly agree (5). The raw data generated from the survey was then computed to an overall SUS 
score ranging from 0 to 100, with higher values indicting a better result. 

4.1.4 Procedure 

Master's students were asked to use the system for one week, and were completely free to decide when 
and for how long to use the system for learning. However, it was mandatory for them to use and to 
answer enough assessment items correctly in order to collect the required points used for their marks. 
Of course, they could also use other resources for their learning in addition to the material provided by 
the Compod system. Previously, accounts for each student had been created separately and sent out to 
them. Furthermore, students got a general explanation of how to use the system and the evaluation 
study in a lecture before starting to work with the system. After one week, the system was closed so 
that the students could not use it anymore. After working with the system, the students were asked to 
fill in the online evaluation survey. 

4.2 Results 

4.2.1 Results of the learning progress and the tool usage 

Since the learning behaviour was tracked by the Compod Service, user model data for each student is 
available. Though this data was used to support the students during their SRL processes, a post-analysis 
revealed interesting information about not only the learning effectiveness of the system, but also 
whether students followed the proposed learning approach. Overall, the data consisted of two types of 
information: learning progress and navigational behaviour. 

Learning progress and knowledge level: The goal of the students was to achieve a score of at least 50 
out of 100 points. In order to earn points, they had to answer assessment items correctly. Having 
available all competences leads to 100 points and a subset of the available competence leads to the 
respective proportion (score = (number of available competences / number of all competences) * 100). 
There was no time limit, so all students had the chance to use the system until they achieved as many 
points as they wanted. 

In the first evaluation round (n=28), 27 students achieved 100 points; one only 62 points. In the second 
evaluation round (n=22), 20 students achieved 100 points, one achieved 92 points, and one dropped out 
after a few minutes of using the system. The positive learning result is an indicator that the students 
were highly engaged in using the system and did not get frustrated (except one or two). This result also 
positively answers the third evaluation question (E3) about whether the learning approach is suitable to 
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learning effectively and to acquiring domain knowledge. 


Navigational behaviour: The user interface consists of four main tools (i.e., goal-setting tool, learning 
tool, assessing tool, and reflection tool) related to different phases of self-regulated learning (see Figure 
7). In principle, students could freely decide which tools they used, how often, and in which sequence. 
They had to complete all assessment items provided by the assessment tool and therefore had to select 
all competences at least once. Analysis of the navigational log data revealed that students had a strong 
tendency to follow the learning cycle "planning-learning-assessment-reflecting." A second smaller 
cycle "learning-assessing" could be observed, which is not surprising since it constitutes a more direct 
way of achieving the required learning score and is a common learning practice in the Austrian 
educational system. An overview of the navigational behaviour is depicted in Figure 7. This diagram 
shows the relative frequencies of the transitions from one phase to another. For example, the 
transitions from the planning phase to the learning phase are 0.85 (or 85%) of all transitions from the 
planning phase. The sum of the relative frequencies from one phase to the other is always 1 (or 100%). 
Additionally, the diagram shows how often the single tools were used by indicating the relative 
frequency of the overall number of visits to the tool. When looking at these relative frequencies in more 
detail, it becomes clear that the learning and assessment phases were visited more often than the 
planning and reflection phase. Data from both evaluation rounds are presented together and separately 
in brackets. 


0.85 



Figure 7. Overview of the navigational behaviour between the SRL tools. The arrows indicate 
navigation from one SRL tool to another and the values indicate the relative frequencies. 


These results also answer the second evaluation question (E2): do learners follow the proposed learning 
approach and do they adopt the self-regulated way of learning? The navigational data shows that 
learners used all the tools, including those related to metacognition (planning and reflection), without 
being forced to and that they freely followed the learning cycle, at least to some extent. Main activities 
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shown between the learning and assessment phase might be explained by the fact that students are 
familiar with this kind of learning, as it is common in the educational context. Interestingly, when 
additionally using the planning and/or reflecting functionality for their learning, students follow the SRL 
cycle, meaning that they first plan, then learn, then assess, and finally reflect on their learning before 
starting again with planning. This behaviour indicates that students adopted the self-regulated way of 
learning by following the proposed learning approach. 

4.2.2 Results of questionnaires 

Usability: The system's usability scored well, with an overall average score of M = 76.01 (SD = 12.85, 
Mdn = 75.00) on a scale ranging from 0 to 100, where higher values indicate a better result. In the first 
evaluation round, a mean usability score of 79.30 (SD = 12.63; Mdn = 80.00) resulted, which indicated a 
good to excellent usability of the Compod environment. In the second evaluation round, a slightly lower 
score resulted with 72.89 (SD = 12.59; Mdn = 72.50). Results are presented in Figure 8. Looking at 
individual items, participants assessed the learnability of the system's functionalities as appropriate, 
meaning that participants need no additional support in order to work effectively in and with this 
environment. The lowest result was for the potential future use of the system; however, the resulting 
score is slightly above the mid-point (M = 2.08; SD = 1.30; Mdn = 2.00), arguing for a satisfactory 
assessment. This result is also reflected in the qualitative feedback given by participants, where most 
students appreciated the system's usefulness for learning. Despite some minor spelling errors, they 
mostly found the system very usable. 
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Figure 8. Overview of results (mean scores and SDs) on the aspect usability for both 
evaluation rounds and total usability score. 


User Acceptance: For assessing user acceptance, ratings on the subscales perceived ease of use, 
perceived usefulness, and intention to use were collected. The results of the mean scores for each 
individual aspect, as well as the overall score for both evaluation phases, are depicted in Figure 9. The 
best result was for perceived ease of use, with M = 6.33 (SD = 0.84, Mdn = 7.00) in the first evaluation 
round and M = 5.58 (SD = 1.50; Mdn = 6.00) in the second. Students found the system generally easy to 
use. The overall score for ease of use, taking into account both evaluation rounds (M = 5.95; SD = 1.27; 
Mdn = 6.00), was strongly correlated with usability: r = 0.65 (p = 0.00). This means that students who 


ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 


121 








JOURNAL OF LEARNING ANALYTICS 


S8LAR 

SOCIETY for LEARNING 
ANALYTICS RESEARCH 

(2015). A Competence-based Service for Supporting Self-Regulated Learning in Virtual Environments. Journal of Learning Analytics, 2(1), 101-133. 

gave the system high marks for usability also gave it high marks for ease of use. For perceived usefulness 
(M = 4.57; SD = 2.03; Mdn = 5.00) and intention to use (M = 4.35; SD = 2.09; Mdn = 4.50), slightly lower 
values could be identified. Perceived usefulness was rated at M = 4.89 (SD = 2.00; Mdn = 5.5) in the first 
evaluation round and M = 4.26 (SD = 2.08; Mdn = 5.00) in the second. Intention to use resulted in a 
mean score of 4.69 (SD = 2.09; Mdn = 5.25) in the first and 4.03 (SD = 2.10; Mdn = 4.00) in the second. 
Overall, this indicates a medium to good result and is completely in line with the qualitative feedback 
given by participants, who highlighted the support for learning that the system can provide. 
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Figure 9. Overview of results (mean scores and SDs) on the user acceptance aspects: perceived ease of 
use, perceived usefulness, and intention to use for both evaluation rounds and in total. 


Learning Approach and System Guidance: In the third questionnaire, students were asked to assess the 
overall learning approach and the guidance facilities provided by the system. Overall, the learning 
approach was rated as moderately good, with a mean score of 5.35 (SD = 1.30; Mdn = 5.67) on a scale 
ranging from 1 to 7 where higher values meant a better result. In the single evaluation rounds, a mean 
score of 5.65 (SD = 1.19; Mdn = 5.67) was obtained in the first round and 5.03 (SD = 1.35; Mdn = 5.33) in 
the second. Looking at the items for both evaluation phases, it became obvious that students found the 
learning approach not only supportive for learning generally, but also provided them with a better 
learning experience than other learning systems. The detailed results of the individual items are 
displayed in Figure 10. These generally positive results were also confirmed by the students when 
explicitly asked about the strengths and weaknesses of the system and its underlying approach. Most of 
them liked the cyclical learning approach, especially the feedback function because it provided a good 
overview of which questions had not been answered correctly. Flowever, they also remarked critically 
that it was not clear at the beginning how to work through these different learning cycles (i.e., planning, 
learning, assessing, and reflecting) and how to use the different functionalities provided by the system, 
especially the visualizations (i.e., clicking on bubbles and marking them with different colours). With this 
in mind, a small tutorial would facilitate dealing with the environment. 
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Figure 10. Results on items assessing the learning approach used by the system. 


Similar results were obtained when assessing the guidance functionality provided by the system. In the 
first evaluation, this aspect was rated at M = 5.56 (SD = 1.12; Mdn = 5.67) and the second at M = 5.25 
(SD = 1.57; Mdn = 5.67). This resulted in an overall score of M = 5.41 (SD = 1.35; Mdn = 5.67) as depicted 
in Figure 11. These positive results indicate that participants see the additional support this guidance 
functionality provides to their learning. They also highlighted the freedom to plan and organize learning 
on their own. However, they also pointed out that a primer on how to use and apply the learning 
approach and guidance functionality would be helpful. 
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Figure 11. Results on items assessing the guidance functionality provided by the system. 


Visualizations: Goal-Setting Tool: Concerning the goal-setting tool, good to medium results were 
obtained for both usability and benefit. For usability, a mean value of M = 5.35 (SD = 0.84; Mdn = 5.60) 
in the first evaluation round and M = 4.79 (SD = 1.12; Mdn = 4.80) in the second were found. Benefit was 
rated slightly lower with M = 4.36 (SD = 1.12; Mdn = 4.29) in the first evaluation and M = 4.43 (SD = 0.88; 
Mdn = 4.29) in the second. When looking at the single subscales, in the first evaluation round scores 
ranged from 4.00 (SD = 1.08; Mdn = 4.00) in the cognitive load scale to 5.47 (SD = 1.05; Mdn = 5.50) in 
the self-descriptiveness scale. In the second evaluation round, similar results were obtained, with mean 
scores ranging from 3.79 (SD = 0.90; Mdn = 3.50) in the cognitive load scale to 4.79 (SD = 1.17; Mdn 
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=4.50) in the suitability scale. Table 4 shows all results in detail. 


Table 4. Results of the subscales, total usability, and total benefit for the goal-setting tool. 



Evaluation Round 1 

Evaluation Round 2 

Total 


Mean (SD) 

Mean (SD) 

Mean (SD) 

Suitability for the task 

5.03 (1.18) 

4.79 (1.17) 

4.91 (1.17) 

Self-descriptiveness 

5.47 (1.05) 

4.63 (1.34) 

5.05 (1.26) 

Total Usability 

5.35 (0.84) 

4.79 (1.12) 

5.07 (1.02) 

Metacognition 

4.61 (1.42) 

4.77 (1.08) 

4.69 (1.12) 

Cognitive Load 

4.00 (1.08) 

3.79 (0.90) 

3.89 (0.99) 

Learning-effectiveness 

4.26 (1.42) 

4.21 (1.22) 

4.24(1.30) 

Total Benefit 

4.36 (1.12) 

4.43 (0.88) 

4.39 (0.99) 


Reflection tool: Similarly to the goal-setting tool, this type of visualization also received good results in 
all aspects (see Table 5). In the first evaluation round, the scores ranged from 4.03 (SD = 0.87; Mdn = 
5.50) for cognitive load to 5.41 (SD = 1.19; Mdn = 5.00) for self-descriptiveness, resulting in an overall 
usability score of 5.25 (SD = 1.22; Mdn = 5.50) and an overall benefit score of 4.19 (SD = 1.21; Mdn= 
4.57). In the second evaluation round, similar results were identified: scores ranged from 3.86 (SD = 
0.92; Mdn = 3.50) for cognitive load to 4.92 (SD = 1.48; Mdn = 4.50) for suitability for the task. The 
overall usability score was M = 4.82 (SD = 1.33; Mdn = 4.50) and the benefit score was M = 4.51 (SD = 
1.05; Mdn = 4.43), also satisfactorily good results. These results indicate that students find the tool 
suitable for reflecting on their learning and see the benefit it can provide. 


Table 5: Results of the subscales, total usability, and total benefit for the reflection tool. 



Evaluation Round 1 

Evaluation Round 2 

Total 


Mean (SD) 

Mean (SD) 

Mean (SD) 

Suitability for the task 

5.09 (1.51) 

4.92 (1.48) 

5.00(1.48) 

Self-descriptiveness 

5.41 (1.19) 

4.72 (1.34) 

5.06(1.30) 

Total Usability 

5.25 (1.22) 

4.82 (1.33) 

5.03 (1.28) 

Metacognition 

4.41 (1.50) 

4.80 (1.35) 

4.61 (1.42) 

Cognitive Load 

4.03 (0.87) 

3.86 (0.92) 

3.94 (0.89) 

Learning-effectiveness 

4.29 (1.51) 

4.39 (1.40) 

4.34(1.43) 

Total Benefit 

4.19 (1.21) 

4.51 (1.05) 

4.36(1.13) 


Summing up the results obtained for both types of visualizations used by the Compod system, generally 
students agreed that visualizations provide useful and relevant information in terms of better planning 
and reflecting on their learning. These results were confirmed in the qualitative feedback. Most 
participants found that the goal-setting visualization provides useful information and is thus suitable for 
planning future learning activities, especially the opportunity to see the dependencies between topics 
(i.e., main topic and sub-topic). Additionally, students appreciated being able to organize their own 
learning process by simply choosing and clicking on their own learning goals. On the other hand, 
however, they pointed out the complexity of the visualization type (i.e., the Hasse Diagram) and 
consequently the need for a short tutorial (e.g., video or animation) explaining the functions (e.g., 
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choosing a bubble, which consequently changes his colour). Regarding the reflection tool, participants 
found it clear, easy to understand, and that it provided good and useful information. 


On a nominal level, the Compod system and services were assessed slightly higher in most qualities in 
the first evaluation study than in the second. The two exceptions were the goal-setting tool and the 
reflection tool, which were rated lower in the first study. However, statistical comparisons within the 
sample (t-tests for independent samples or, respectively, non-parametric tests where necessary) 
between the evaluations of the two student cohorts yielded no significant differences. 

5 DISCUSSION, LIMITATIONS, AND OPPORTUNITIES 
5.1 Discussion 

Referring to our original research questions, the main aim was to create a framework composed of 
different learning methodologies (R2) and system implementing this framework (R3) that supports 
learners to follow the self-regulated learning paradigm and to acquire knowledge of the subject domain 
(Rl). The proposed solution to these research questions was described in our psycho-pedagogical 
framework that incorporates Competence-based Knowledge Space Theory, personalization approaches, 
the Open Learner Model, and learning analytics methods, in order to provide personalized guidance in a 
self-regulated learning process. In order to validate the proposed solutions, a study, guided by four 
evaluation questions was undertaken to give insight into the extent to which this framework and system 
answer the research questions. In this evaluation, a descriptive study design using a multi-method 
approach for gathering data was used in order to gain in-depth data on the usefulness and applicability 
of the Compod system and its different functionalities. To ensure the same learning conditions for each 
learner in the context of a university course where students finished with a certificate, no control group 
was established. Because of the chosen design, the study also did not include a pre-test-post-test 
condition. This might be problematic, especially when making conclusions about the learning 
effectiveness of the system. However, the focus of this formative evaluation, as well as discerning what 
was good and useful to students, was on identifying any issues or potential problems with the 
technology and the underlying learning approach from a user-centred perspective. Such a procedure 
should ensure that valuable information for further improvement of the software is provided to the 
developer. 


The first evaluation question (El) addresses the usability and user acceptance of the overall system and 
approach. The results from questionnaires were rather good on both user acceptance and usability. This 
indicates that, in general, the system and its approach are acceptable. The second evaluation question 
(E2) specifically targets the learning approach and thus the framework. The results from these 
questionnaires indicated that students like this way of learning. The third evaluation question (E3) asked 
if learners actually acquire domain knowledge with this learning approach. Results retrieved from the 
user-model data show that learners achieved the goal of mastering the subject domain. Almost all 
students achieved the maximum score even though this was not required. Finally, the forth evaluation 
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question (E4) specifically targets the goal setting and reflection tools and their visualizations. Results 
indicate that usability and the benefit for the learning process is above average, and students explicitly 
pointed out the positive effects in the qualitative feedback. 

Overall, the results of the study are quite promising, indicating that the elaborated framework and 
developed system fulfil their purpose. Individual results on the evaluation questions point out that 
students accepted the system with its functions, user interface, and pedagogical framework. 
Additionally, the learning results in terms of domain knowledge were successful. If the system (R3) and 
the framework (R2) are suitable for self-regulated learning and achieving the learning goals (Rl), then 
the Compod system passed this test. 

Another discussion point addresses the system's relation to similar systems in the educational area, 
which already has a long tradition. The Compod system and approach follow the tradition of adaptive 
systems. It uses the same types of models (learner, domain, tutoring, and user interface model) and 
follows the Open Learner Model (OLM) approach by making these models visually accessible to the 
learner. Some similar systems deal with concepts, concept hierarchies, and concept-learning object 
relations. For example, both Interbook and AHA! present learning objects and concepts and give the 
user some freedom to navigate through the concept space (De Bra, Santic, & Brusilovsky, 2003). 
However, Compod has at least two distinguishing aspects. First, Compod is built upon CbKST, which not 
only includes concepts and learning objects, but also assessment items and probabilistic methods for 
adaptive testing and learner model updates, and brings them together in a holistic theory. Though not 
all features of CbKST are now used in Compod, there is still an opportunity to integrate them into the 
system. Second, Compod uses explicit support of self-regulated learning in terms of a phased approach 
that stimulates metacognitive activities (e.g., goal setting, planning, reflecting, and self-evaluation). A 
prominent example for an adaptive system using Knowledge Space Theory is the ALEKS system 
(Falmagne, Cosyn, Doignon, & Thiery 2006). ALEKS uses an adaptation strategy based on Knowledge 
Space Theory (without the competence-based extension) to perform efficient, adaptive knowledge tests 
and personalized learning paths. However, the adaptation strategy and user models are not open to the 
learners. 

5.2 Limitations 

Though the overall feedback from the user study was rather positive, there are also some limitations to 
this approach. Self-regulated learning is a complex process and it is very unlikely that students can 
increase their SRL capabilities in a course unit during one week. Though this was not the aim of the 
study, it is still worth a closer look. Despite the fact that self-regulated learning is a complex process, it 
can also be seen as an integrated process involving the active construction of new behaviour affecting 
learning. In this way, self-regulated learning can be seen as a constructive learning process. Research has 
shown that the active application and development of self-regulatory processes is essential for 
enhancing self-regulatory skills (e.g., Pintrich & Zusho, 2002). Such a constructive view of learning 
understands the learner as being actively, mindfully, and effortfully involved in the learning process, or 
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more concretely, in the process of knowledge and skill acquisition. With this constructive learning theory 
in mind, it becomes clear that even if students did not significantly enhance their self-regulated learning 
skills within the week of the study, applying them was still valuable training. However, in this study, the 
research focus was not on investigating whether self-regulated learning skills are enhanced by the 
Compod system — the research focus was more on investigating whether the learning approach itself 
applied. For future work, having a concrete focus on the development of students' SRL skills and their 
development by utilizing a standardized strategy to evaluate an increase of these competences would 
be useful (e.g., Hadwin, Nesbit, Jamieson-Noel, Code, & Winne, 2007; Pintrich et. al, 1987). 

Another problem with the evaluation was that a few learners mastered the unit for the evaluation not 
alone but in a small group. By analyzing the time stamps of the navigation events, it seems that some 
learners did the learning process together. However, doing assessment items together and probably not 
learning the content as suggested brings a slight distortion of the results. Since this is an online course, it 
is hard to prevent such effects. On the other hand, this also indicates that at least some students prefer 
to learn in groups, which reveals a general shortcoming in the presented approach, which is the missing 
aspect of collaborative learning. As known from the literature on SRL, collaborative learning provides 
opportunities for metacognition, motivation, and self-regulated learning in general so there is much 
room for extending the presented approach with collaborative aspects. For example, groups could be 
formed automatically with the same competences or peers could be recommended who have already 
mastered competences with which other students have problems. 

Finally, using complex visualizations as a basis for planning, goal setting, and reflecting can cause 
problems if a learner does not understand such visualizations. In our user study this problem did not 
occur but this is not surprising since computer science students understand a graph-based view. 
However, other learners might have difficulties understanding graph visualization, which would lead to 
completely different results. One solution would be a short training session for using these 
visualizations. 

5.3 Opportunities 

Self-regulated learning is a complex psychological and pedagogical matter and appropriate technological 
support is a huge challenge. The presented Compod system supports self-regulated learning to some 
extent, but still leaves room for much improvement and further in-depth research. This section 
discusses some of the further research and development opportunities. 

A key aspect of self-regulated learning is related to the cognitive and metacognitive activities and the 
sequences in which a learner performs them. When using technology for learning, these activities are 
manifest in interactions with the user interface. The sequences of these interactions can be subsumed 
as navigation behaviour and thus become an interesting and valuable resource for further research. 
Data collected in the user model contains information on the selected goals in terms of competences, 
the visited learning objects, the answered assessment items, and the selected tools. This information 
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brings further research opportunities by analyzing in depth the navigation behaviour through these 
elements; for example, a detailed analysis of the extent to which a learner has followed the prerequisite 
structure provided in the competence map. This would provide insight, if this type of scaffold were 
accepted. Relatedly, analyzing how many competences per goal and how much time per goal (visited 
learning objects, assessment items) would facilitate investigating any correlations between the 
navigation behaviour and the learning progress. An analysis of the navigation behaviour through the 
tools has already been made (see Figure 7). However, there are more possibilities to relate the tool 
navigation behaviour with the goals, learning objects, and assessment items; for example, if there is any 
correlation between the navigation patterns and the learning outcomes. 

These further analyses could in turn be used for live feedback on the reflection tool (Figure 6). Currently 
the reflection tool just shows information on the learning progress in terms of mastered assessment 
items and acquired skills. However, information on the navigation patterns could be used as a basis for 
further feedback. For example, the degree to which extent the learner follows the prerequisite structure 
of the competences could be visually displayed. We argue that it is pedagogically meaningful to follow 
this structure, as this feedback would reinforce the adoption of respective learning paths. In addition, 
whether the navigation behaviour correlates with the learning progress could be interesting for learners. 
This would require further evaluation and analysis to see if it affects learning. 

The domain model (competences, learning objects, assessment items) used in the evaluation was rather 
small; larger domain models with more learning objects and assessment items would be beneficial for 
the learner. In the current domain model, only one learning object and one assessment item are 
available for each competence. However, the Compod approach allows for defining many more learning 
objects and assessment items for each competence. Learners might benefit from multiple perspectives, 
explanations, and examples to train a single competence. Also testing competences would benefit from 
multiple assessment items, which can foster the accuracy of the result. 

The evaluation itself mainly addressed the question of whether the Compod system and its pedagogical 
approach would be accepted by learners; however, many more aspects can be evaluated in further 
studies. While the present evaluation included only one lecture and the average use of the system was a 
few hours, a longer study over the whole semester of following an experimental design could reveal 
more insight into self-regulated learning behaviour and its effect on learning outcomes. The whole 
course studied consists of about ten lectures and each of the lectures could be adapted for the Compod 
system. A test on self-regulated learning skills could be made at the beginning and end of the semester, 
which would show whether there were improvements over this time. Furthermore, an analysis could be 
made of whether learners change their learning behaviour over time in terms of their navigation 
behaviour. Using pre- and post-knowledge tests could also reveal if the frequent use of Compod and 
self-regulated learning approaches has a positive effect on learning outcomes. 

Finally, due to the modular technical approach (separation of front-end and back-end), there is the 
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potential to try out different user interfaces and thus modified SRL approaches. This might lead to 
designs with different levels of guidance, different amounts of feedback and recommendations, and 
different types of personalization. There is already an alternate user interface design available that 
reduces the amount of freedom in the goal-setting phase (Kopeinik et al., 2014). This modification was 
designed for vocational training with workers who are not used to setting their own learning goals. 

6 CONCLUSION AND OUTLOOK 

This paper has presented our framework for self-regulated learning. We identified a self-regulated 
learning process model and developed scaffolding techniques on a conceptual and technical level to 
support cognitive and metacognitive activities. We evaluated the learning approach and the technology 
in a descriptive evaluation study with computer science students. Students reacted positively and 
tended to accept both the learning approach and the system. 

In the authors' opinion, the most innovative aspect is the combination of different learning concepts, 
which leads to a new framework that addresses learning support on both cognitive and metacognitive 
levels. While navigating the learning resources and attaining respective domain knowledge is related to 
the cognitive level, control of the overall learning process, including goal setting and reflecting, is related 
to the metacognitive level. The framework has means to capture both cognitive and metacognitive 
learning activities and uses them for personalized support, including goal setting, navigation, and 
reflection, on both the cognitive and metacognitive levels. In this way, broader, more holistic support for 
self-regulated learning is provided. To the best of our knowledge, a system that combines self-regulated 
learning with its metacognitive aspects and learning on the domain level is not available so far. 

This paper contributes to the learning analytics field, as the presented framework and learning support 
strategies are based on traced and monitored learner data. Captured learner data include selected 
learning goals, visited learning objects, solved assessment items, or the navigation behaviour through 
the learning cycle. This information is used to provide recommendations and visual feedback following 
the idea of OLMs, in order to support self-regulated learning. In this way, learning analytics techniques 
together with OLM are used to assist a learner pedagogically in controlling his own learning process. 

Future work includes more and deeper analysis of the learner interaction data in order to use it for 
further guidance mechanisms; for example, the navigation behaviour through the competence structure 
and related learning objects could be further analyzed. Having the nature of prerequisite relations in 
mind, the meaningfulness of the learners' navigation behaviour can be seen on different levels. As well, 
the reflection tool can benefit from further analysis in terms of learning behaviour. For example, analysis 
of the navigation behaviour presented in Figure 7 could be made visible in the reflection tool. This would 
feedback on one's own learning process to exploit learning analytics methods further while using the 
system. Another type of improvement includes collaborative support; for example, information can be 
provided about peer learners, so that learners can compare themselves with others regarding learning 
progress or navigation behaviour. 
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In future research, a specific focus could be on further investigating whether the Compod system with 
its proposed learning approach has a positive effect on learning in general, and on the enhancement of 
competences in self-regulated learning in particular. Thus we are planning to alter our experimental 
design to including a pre-test-post-test and a control group in order to investigate the cognitive 
processes involved in the learning experience in more detail. 
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